A virtual environment (VE) is an array of sensory cues generated in response to a user's actions that gives the user the impression of dealing directly with a three dimensional model. Typically either a head mounted or fixed display is used to present a dynamic perspective image of the visual environment as the user moves within the VE. The virtual environment may be synthetically generated within a computer or may involve sensory input from a remote physical location (as in tele-operations). Virtual locomotion is movement over long distances in a virtual environment which is controlled by a user remaining within a relatively small physical space.
An avatar is a computer model of an articulated human figure that depicts the user's body in the virtual environment. The motion of the avatar is controlled by the user. When the user has a first-person view he sees the virtual world through the eyes of the avatar, so that the view of the VE changes as the avatar's head turns and translates through the VE. The major components of the avatar's motion involves locomotion: orienting and moving the body over the terrain. This will be the primary focus of the controls described herein. The new approach to controlling the avatar's motion also addresses turning the head to look around, and adopting different stances and postures. These secondary aspects of motion will be addressed after control over the major aspects of locomotion has been covered.
Virtual locomotion can be used for a variety of purposes, such as: training or rehearsal in executing skills, tasks, strategies, and navigation that involve moving through an environment on foot; planning activities that involve moving through a target environment; evaluating the ergonomics or aesthetics of structures designed for human habitation, or of devices intended for mobile operation; piloting remote surface vehicles; communications between people at different locations when they want to relate to each other in a way that involves locomotion; and entertainment experiences that involve moving through a virtual environment. Particular applications would include training people to perform hazardous tasks without risking bodily harm or to train soldiers in combat simulators where the soldier interacts with the surrounding environment which includes other members of a team.
We adopt terms used in both vehicular and human navigation literature to describe different types of locomotion. When focusing on locomotion we will assume, in order to facilitate our explanation of the invention but without being expressly limited by the depictions in the figures, that the head remains level, faces straight-ahead, and maintains a fixed alignment with the rest of the upper-body as pictured in the figures. There are three components of motion in the horizontal plane. Heading is the angular direction the upper-body faces; course is the angular direction the pelvis translates; and displacement is the distance the pelvis translates along the course. The first two components specify orientation with respect to a fixed reference direction and the third specifies distance traveled. The alignment-offset is the angle between the course and heading. The angle between the direction of a step (course) and the direction of the upper body (heading) determines the kind of step being taken (FIG. 1). A figure or avatar 1 representing a person in a virtual environment is depicted from above with arms extended holding a rifle. Accordingly, the rifle is shown aligned in the direction the avatar 1 is facing, in FIG. 1 and in subsequent figures (unless indicated otherwise), such that avatar 1's heading is aligned in the direction of the held rifle while the bold-face arrow under avatar 1 indicates its course.
The application of course, heading and displacement yields (at any point in time) a position and orientation in 3 degrees-of-freedom (DoF): the X,Y position on the ground and a Yaw rotation about the vertical axis of the global coordinate frame. (Degrees-of-Freedom (DoF) is defined as the number of independent variables describing the state of a system or device. A basic computer mouse, for example, provides 2-DoF since it reports continuous X and Y displacements as it moves over a surface.)
Classification of Human Locomotion
We have developed a means of classifying the kinds of motion a person can make while walking or running in the real world in terms of how course and heading vary. This new classification of motion is useful for characterizing existing systems as well as for explaining the new collection of open loop controls of the invention as elaborated on below. All the kinds of motion shown in FIG. 2 involve avatar 1 moving along a path, with the consecutive positions of avatar 1 for each of three positions indicated sequentially by the direction of the arrow alongside, and again (as in FIG. 1) with the avatar 1's heading in each instance being aligned with the rifle while the bold-face arrow under avatar 1 indicates its course (here and also in subsequent figures). Although a person can also turn the heading without translating by twisting the pelvis or stepping to turn in place, those motions are not depicted. We define four classes of motion. Steering Motion: Course and heading remain co-aligned while moving along a path (FIG. 2, upper left). Steering motion can be used to move directly towards a stationary or moving target and to follow a route marked out on the ground by facing forward along the trail. Canted Motion: Course and heading maintain a fixed alignment-offset (defined above) while moving along a path (FIG. 2, upper right). Steering motion is a sub-class of canted motion. Note that stepping backward along a path is also an example of canted motion (with a fixed alignment-offset of 180 degrees), but may also be thought of as steering in reverse. Oblique Motion: Heading remains in a fixed direction while moving along a path (FIG. 2, lower left). Notice that moving along a straight path with a fixed alignment-offset qualifies as both canted and oblique motion (as shown on the left in each respective panel), so the classes of motion partially overlap. Scanning Motion: Heading is free to turn separately from the course while moving along a path (FIG. 2, lower right). Scanning motion can be used to search from side-to-side while moving or to remain directed towards a stationary or moving target, while traversing a straight or curved path. Scanning while traversing a curved path is the only case where both the heading and course vary independently.
Definitions Relating to Viewing
A virtual locomotion control directs the user's avatar through the virtual world and determines, at any given point in time, its situated-heading. The situated-heading is the global 4-DoF (X, Y, Z, Yaw) position and orientation of the avatar's head when the avatar is in an upright stance looking straight ahead. (There are actually only 3 degrees of freedom in the situated-heading since the Z coordinate is not under the user's control, but only varies with the height of the avatar.) Relative to its situated-heading, the avatar may be directed to crouch, lean, look up, down or to the side. The head-pose is the resultant 6-DoF translation and rotation of the avatar's head relative to the situated-heading. The head-pose combines with the situated-heading to yield the avatar's viewpoint, which is the global 6-DoF displacement and orientation of the avatar's head in the virtual world. The viewpoint determines the user's first-person view of the virtual world as seen through the eyes of the avatar.
When controlling a remote vehicle, the avatar's head depicted in the accompanying figures corresponds to the remote vehicle's turret (where control cameras are mounted that provide images of the actual environment in which the remote controlled vehicle exists). Thus, the situated-heading is the global 4-DoF position and orientation of the vehicle's turret when the turret is aligned with the vehicle's chassis. The head-pose is the 6-DoF translation and rotation of the turret relative to the situated heading, and the head-pose is combined with the situated-heading to yield the vehicle's viewpoint. The viewpoint determines the user's view of the virtual world as seen through the lenses of one or more cameras attached to the vehicle's turret.
Definitions Relating to Controls
The following terms are used to describe controls. Input Control Device: a physical mechanism a user moves or applies forces to. The movement or forces are sensed, digitized, and transmitted to a computer. Control Mapping: the transformation of control input to the resulting motion of a controlled element (i.e., all or part of a vehicle or avatar). Turning the steering wheel of an automobile, for example, turns the orientation of a set of tires which turns a vehicle in motion. Rate Control: a control that maps the position or displacement of (or force applied to) an input device to the speed of a controlled element. There are rate controls over translation and rotation. Positional Control: a control that maps the position or displacement of (or force applied to) an input device to the position of a controlled element. Position may refer to either translation or rotation. For example, translating an input device might alter the rotation of a controlled element. Translational Control: a positional control which controls translation. Directional Control: a positional control which controls rotation.
The following additional terms are used to further describe controls. Closed Loop Control: a control that requires additional sensory feedback (to that described for “Open Loop Control”, below) for the user to determine how far a controlled element has turned or translated. Joysticks used to control the rate of turning the heading provide a good example of a closed loop control. The user pushes a joystick to one side and then watches the display to determine how far the view has turned. (In theory, if the exact time sequence with which the joystick is deflected is known and the mapping between the deflection and rate of turning is known, it is possible to determine how far the view has turned; but in practice people rely on visual feedback, since their ability to estimate the extent of turning is limited when deprived of visual (or auditory) feedback.) Open Loop Control: a control in which the user is aware of the extent of a translation or rotation of a controlled element (e.g. an avatar or a remote controlled vehicle) as he enters it with the input device. The user feels the position of the input device (or force applied to it) to sense the turn or translation without the need of further visual or auditory feedback in response to his action. Open loop control is therefore independent of external feedback such as visual or auditory cues but benefits from internal sensory feedback (e.g., the sense of touch). Open loop controls are typically positional controls with a constant mapping between the distance, angle, or force applied to the input device and the resulting effect. A positional control is not necessarily an open loop control or at least solely open loop. For example, a relative positional control (like a mouse) provides open loop control over the change in position but only provides closed loop control over absolute position (of a cursor on the screen).
The term tracker as used herein includes the following: A tracking system made up of a set of components that allow a position detector to determine one or more of the spatial coordinates (X, Y, Z, Yaw, Pitch, Roll) of one or more tracked elements (e.g., parts of the body). Many tracking systems require the attachment of one or more tracking components to the tracked element. A “sensor” is a device that registers specific forms of structured energy. Position detectors take in readings from one or more sensors to determine the coordinates of tracked elements. In the case of a conventional optical tracking system, a set of markers is attached to the element to be tracked, visible to optical sensors mounted in the room. Other tracking technologies, including acoustic and potentially Radio-Frequency tracking, may adopt a similar configuration. One or more emitting or reflective elements for the sensed form of transmitted energy is attached to the tracked element and sensors are placed in the room. Another configuration is to place the sensor(s) on the element to be tracked and place emitting or reflective elements as points of reference in the room. Some sensors can detect their own motion without the need for reference markers: Inertial sensors can register translational or rotational accelerations which can be integrated (twice) over time to determine translational or rotational displacements. Inclinometers use the earth's gravity to sense tilt. Magnotometers can register their orientation with respect to the earth's magnetic field. Such sensors are mounted directly on the tracked element. Another approach is to use optical sensors to track with, but to use the natural visual features as reference points, instead of applying a special set of markers (emitters or reflectors). In that case the optical sensors could either be mounted in the room with the receiving aperture directed towards the tracked element, or the optical sensors could be mounted on the tracked element and look for visual features of the surrounding room to sense its own motion. A tracker can also employ instrumented mechanical linkages to connect the tracked element to a known frame of reference. The change in the relative positions of the linkages are sensed and those readings are fed into a position detector.
Vehicular Controls
A wide range of controls for ground vehicles exist. Different kinds of vehicles often use different controls, but this is not always the case. Automobiles and trucks typically use steering wheels, foot operated brakes and accelerators, and a means of shifting the transmission. Motorcycles and bicycles use handle-bars for steering, with hand-brakes and shift controls mounted on the handle-bars. The controls of vehicles are often directly tied to the physical mechanisms used to impart motion or redirect the motion of the vehicle. In a car the accelerator increases the power generated by the engine, the brakes apply friction to stop the wheels from turning, the steering wheel turns a pair of the car's wheels. The driving controls give the user fairly direct control over the operation of the vehicle. Such an approach tends to bias the nature of the controls in a number of ways. Often rate controls are adopted because that is how the mechanics of the vehicle works. An automobile's steering wheel provides a rate control over turning. An automobile continues turning so long as the steering wheel remains turned to the side. A second bias is that many vehicles only support steering motion. Thus with a car the course and heading remain co-aligned while the car advances, except when the contact between the wheels and ground slips and the vehicle slides across the surface. There are exceptions, however, such as a tank with a turret that rotates independently of the main chassis. Since one can associate heading with the alignment of the turret, the “upper body” of a tank, while the course aligns with the main chassis, the two can turn independently. Treaded vehicles like tanks and some tractors also have treads on each side that may be rotated independently. Turning the treads in opposite directions turns the vehicle in place, unlike an automobile.
Contrasting Vehicular Controls with Natural Human Bipedal Locomotion
The controls used to drive ground vehicles do not provide as wide a range of maneuvers as are readily achieved using foot-based locomotion. The human body is controlled differently. There is no single part of the body that corresponds to the main chassis of a vehicle. Different parts of the human body can turn and translate relative to one another. Different actions can contribute to turning the heading: When standing, the legs can swivel at the hip and ankle joints to turn the pelvis and the rest of the upper body relative to the feet. Pivoting the feet on the ground or stepping to turn turns the feet with respect to the ground. The primary means of displacement (displacing the body) via natural locomotion is by stepping. The feet are successively raised and translated above the ground as steps are taken. This translates the body over extended distances. Shifting the pelvis between the support feet provides additional local displacement. Humans also have a direct vestibular sense of how much the head has turned, an internal, proprioceptive sense of the position of every part of the body, and a central nervous system that integrates all this information to allow a person to maintain their course even though different parts of the body may turn in different directions to achieve different effects. A man on foot is more maneuverable than a tank: Even though a tank may control the heading of its turret independently of its course, a tank cannot change its course as abruptly as a man can (e.g., to step to the side); a tank must first turn its main chassis to move in a new direction.
Conventional Controls Used for Virtual Locomotion
Controlling the motion of the user's avatar or vehicle is a key element of a wide variety of computer games and simulations. A variety of controls are available for that purpose. This section describes commonly used control devices and the conventional control mappings which are applied to their inputs.
2-DoF Joystick
We classify a typical joystick as a 2-DoF centering input control device. The normal orientation of a spring centered joystick (JS) is vertically upright, from the surface it is mounted on. The joystick may be pushed off center in any direction. The direction of a joystick can be specified by a yaw angle (analogous to a compass heading). The deflection of a joystick is how far it is pushed in any direction from being vertically upright. When a joystick is released it springs back to an upright centered position. Another way of describing a joystick's motion is to say that it may be deflected to either side of two orthogonal (X-Y) axes, with one axis running front-to-back and the other side-to-side. Basic joysticks provide two DoF which can be considered either as the direction and deflection of the stick or as two orthogonal axes of deflection.
Joysticks have conventionally been used as rate control devices. They are also frequently treated as dual control devices in which the two orthogonal axes control two independent variables. When controlling motion, they are typically linked to two different aspects of motion, such as the rate of turning and the speed of translation.
In the early days of 3D computer games, a single joystick was used to move an avatar about through a computer generated landscape. The deflection of the joystick along its front-to-back axis sets the avatar's rate of displacement. Pushing the JS straight forward moves the avatar straight ahead; pulling the JS straight back reverses the motion; centering the JS stops the avatar. The further the JS is pushed forward or back, the faster the avatar translates in that direction, thus the front-to-back axis of the joystick provides rate control over displacement. The deflection of the joystick along its left-to-right axis sets the rate at which the course and heading turn together, thus the left-to-right axis provides rate control over steering. Pushing the JS to the left steers to the left; pressing the JS to the right steers to the right. The further the JS is pushed to either side, the faster the course and heading turn together in that direction. The heading always faces in the direction of forward motion, while the avatar moves either straight ahead or in reverse. The course and heading turn together in place (without translation) when the JS is pushed exclusively to one side or the other. Pushing the JS in a diagonal direction engages both steering and translation, so the avatar turns while advancing (or retreating). This control mapping does not support oblique, canted, or scanning motion since course and heading remain co-aligned. Additional input is required to provide access to that range of motion.
Gamepad: Dual 2-DoF Joysticks
Gamepads are widely used to control virtual locomotion in many first-person shooter games, especially on game consoles. Referring now to FIG. 3, a gamepad 2 provides a pair of 2-DoF thumb-joysticks 3 and 4 used to change the course, heading, and displacement of the user's avatar. The controls are applied in a fairly standard manner. In the right-hand figure, the deflection of the right joystick 3 along its left-to-right axis sets the rate at which the course and heading turn together, providing rate control over steering. (Deflecting this joystick along its front-to-back axis sets the rate at which the head-pose pitches up or down, but is not part of locomotion.) We call the right joystick 3 the steering joystick (FIG. 3, right). Pushing the steering joystick to its full leftmost extent turns course and heading together to the left at a maximum angular rate (degrees per second). Pushing it only partially over to the right turns them to the right at a slower rate.
The left joystick 4 (note—the abbreviation “JS” will be used interchangeably for the term “joystick” herein) is typically considered to be a two-axis (X-Y) rate control over X-Y translation in a reference frame that turns with the heading, as shown in the left-hand figure of FIG. 3. There is, however, a functionally equivalent interpretation that factors its control over the course and displacement (and fits in better with the new approach presented in the description of the invention, below). We prefer to describe the operation of this JS in terms of polar coordinates (R-Theta): The deflection (R) of the left joystick in any direction sets the rate of displacement. The angular direction (Theta) in which the left joystick is pushed determines the direction of the course relative to the current heading (FIG. 3, left). In general, we call the left joystick 4 the offset joystick because it controls the alignment-offset in a variety of different control mappings of dual joystick gamepads. In particular, we call the angular offset of the course with respect to the heading the course-offset. With this interpretation the offset joystick provides directional control over the course-offset and rate control over displacement. When the offset joystick is solely engaged, the course is redirected and the heading remains unchanged. FIG. 1 above shows examples of alignment-offsets in terms of stepping patterns: Pushing the offset joystick forward directs the course forward (forward step); pushing it left or right moves the course to the side relative to the heading (side step); pushing it back directs the course back (back step); and pushing it on the diagonal produces a diagonal course relative to the heading (diagonal step). Pushing only the offset joystick without engaging the steering joystick (i.e. joystick 3) moves the avatar in a straight trajectory in the specified direction; deflecting the steering joystick without engaging the offset joystick turns the avatar in place. In summary, the offset joystick redirects the course without altering the heading. The steering joystick controls the rate at which the course and heading turn together. Therefore, we categorize the conventional gamepad control mapping as a course-offset control mapping, with rate control over steering and directional control over course-offset.
The conventional gamepad control mapping described above is typical for controlling first-person shooter games. In addition, it is common for games to add additional control mappings which allow the user to assign different functions, polarities, ranges, and scales to the joysticks and buttons provided by gamepads. These currently available options do not include the new forms of open loop controls described in this invention.
Executing Different Classes of Motion with the Conventional Gamepad Control Mapping
The steering and offset joysticks are manipulated (control action) to move the user's avatar through the virtual world (control effect). Because of the way the joystick actions control course, some classes of motion are easier to achieve than others. Moreover, redundant control over course gives rise to a multitude of different ways of entering the same final course.
Steering motion requires deflecting the steering joystick while pushing the offset joystick forward. The situated-heading changes as the heading turns and the avatar advances along the trail. Steering motion is controlled in a closed loop manner because the user relies on visual feedback to determine the extent of rotation and translation. Both steering and displacement are rate controlled. Oblique motion is accomplished using only the offset joystick. Heading remains fixed in the current direction because no steering occurs. The user enters the course simply by pushing the offset joystick in the desired direction without need of visual feedback. Therefore, oblique motion is controlled in an open loop manner. The user continually feels the current direction of the course-offset. Scanning motion following a straight path is difficult using conventional gamepad controls. Referring now to FIG. 4, the steering joystick must turn the heading to scan, but the course turns along with it (FIG. 4, left). To follow a straight path (FIG. 4, right), the offset joystick must counterturn to cancel out this rotation of the course (FIG. 4, middle), as indicated by “summing” the left and middle figures to obtain or equal the motion shown in the right figure. It is difficult to balance the two forms of turning since one joystick is controlled by rate and the other by direction. Without knowing how far the steering joystick changes the course direction, the user does not know where to point the offset joystick to counter it. Scanning motion following a curved path is even more difficult using such controls.
Conventional gamepad controls favor visually directed motion because rate-based steering relies heavily on visual feedback. Both canted and scanning motions allow the avatar to turn to face a target while moving along a path. Users compensate for the weakness in executing scanning motion by relying heavily on canted motion. They adopt a control strategy called targeted canted motion in which the heading is constantly turned toward the target while the alignment-offset of the course and heading remains fixed in order to direct them together as a unit. Targeted canted motion is performed by deflecting the steering joystick while holding the offset joystick in a fixed direction. A family of spiral trajectories emerges by doing so, as shown in FIG. 5. A sophisticated user can smoothly transition between these trajectories to achieve even more complicated paths. The term strafing is used in the gaming community to denote oblique motion, shooting while moving obliquely, and the form of targeted canted motion just described. Circle-strafing is the tactic of circling a target while facing toward it.
Dual 2-DoF Joystick Control of Turreted Vehicles
A hybrid control mapping exists for controlling tanks in simulation games and remotely controlled turreted vehicles. It controls the main chassis using one joystick and turns the turret and elevates the gun with the second joystick. The JS controlling the main chassis works as described above regarding how a single 2-DoF joystick is used to drive an avatar in early 3D computer games: Front-to-back movement of this JS controls the speed of the tank while side-to-side motion of the JS steers the tank chassis. The other JS controlling the turret and gun elevation operates as a rate control over the turret and gun elevation: Deflection of the joystick along its left-to-right axis sets the rate at which the turret revolves; deflection of the joystick along its front-to-back axis sets the rate at which the tank's main gun elevates, pitching up or down.
Since the heading of the tank corresponds to the direction in which the turret faces, this dual joystick control provides two ways of turning the heading: (1) The turret is carried along as the chassis turns under it, and (2) the second JS allows the turret to turn with respect to the chassis. The tank's course remains aligned with the chassis because the tank translates along the treads suspended from the chassis. The left-to-right axis of the second JS provides rate control over turning the heading-offset, i.e. the angular offset of the heading with respect to the course. (The front-to-back axis of the second JS provides rate control over the gun's pitch, but this is not part of locomotion.) In terms of a tank, the heading-offset corresponds to the orientation of the turret with respect to the tank's chassis. Heading-offset control exerts a different kind of control than the more widely used course-offset control previously described for controlling human locomotion (see FIG. 3), where the second JS provides control over the course-offset.
Heading-offset control makes it easy to execute scanning movement while moving along a straight path. Since the turret can turn independently of the main chassis, the user can maintain the course with one JS while turning the heading of the turret with the other JS to scan in different directions. The tank continues to move in a straight path while the turret turns to look about. The advantages of heading-offset control will be further described in the description of the invention below.
Because it is rate-based, this type of heading-offset control has a major problem: the user loses track of the course when both JS controls are adjusted at once. The heading turns both during steering and scanning (using both the steering JS and the heading-offset JS). This redundant means of turning the heading can be confusing. Since both steering and heading-offset turns are performed via rate control, this approach provides closed loop control over both course and heading. Thus the user must rely on the changing visual scene to judge how far his heading has turned. If the user only steers the chassis or turns the turret (heading-offset)—one at a time—the extent of each heading change can be visually estimated. Even then, the user will have to remember how much the heading has turned separately from the course, because there is no information available from a static first person view, looking in the direction the turret is facing, to tell the user how the heading is aligned with the course (i.e., how the turret is oriented on the tank chassis). If the chassis turns in place to redirect the course while the turret simultaneously counter-turns to retain the heading direction, there is no visual indication from a first-person view how far the chassis has turned. The user loses track of where he will go when the steering JS is pushed straight ahead to ‘advance’. If the tank is translating and the turret has stopped turning the course can be inferred from optic flow. The center of outward optic flow specifies the direction of locomotion. If this point is within the visible field-of-view the user will perceive the tank is moving towards it.
This control mapping makes accurate control over oblique movement difficult. There is no reliable way of executing a side-step movement. If the user counter-turns the heading-offset JS to keep the heading facing forward while steering to go sideways he will not know when the course has reached the desired 90° orientation.
Although a straight course can be maintained while scanning, it is difficult to scan following a curved path since it involves turning the chassis and turret at the same time. In so doing the user loses track of the course. Awareness of one's course and heading is vital. Without it the user cannot make full use of the tactical movement capabilities of heading-offset control.
3-DoF Joystick
Some joysticks provide a third DoF by allowing the user to twist the shaft of the JS to turn it in either direction. Such JS are spring-loaded to return the shaft back to a neutral (centered) position upon release. This third axis typically provides a rate control over steering. The more the shaft is twisted in one direction the faster the course and heading turn together in that direction. That assignment frees up the side-to-side motion of the JS to control other aspects of motion.
The side-to-side and forward-and-back deflections of the joystick provide a two-axis (X-Y) rate control over X-Y translation in a reference frame that turns with the heading. This is equivalent to the mapping of the offset joystick of a conventional dual joystick gamepad used to control the avatar's locomotion. As for that mapping, the deflections of the JS may be described in terms of polar coordinates (R-Theta): The deflection (R) of the JS in any direction sets the rate of displacement. The angular direction (Theta) in which the JS is pushed determines the direction of the course relative to the current heading (i.e., the angular offset of the course with respect to the heading). The movement of the joystick thus provides directional control over the course-offset and rate control over displacement.
The 3-DoF JS uses a control mapping similar to that used with the conventional dual joystick gamepad. It is a course-offset control mapping, with rate control over steering and directional control over course-offset. Since scanning motion is difficult to control, it also relies heavily on targeted canted motion for turning the avatar to face a target while moving along a path.
Keyboard and Mouse Controls
The mouse is an input device that provides a 2-DoF relative positional input. The mouse is active when in contact with the surface it strokes over. The input from the mouse does not change when it is lifted and repositioned, making the mouse a relative translational input device. The input of a mouse can be mapped in different ways to control different things. Mice are often used to steer a cursor around on a display screen. In that case a 2-DoF relative translational input is used to control a 2-DoF absolute translational output. The motion of the mouse may be rescaled to drive the motion of the cursor. This scaling is referred to as the control-to-display (CD) ratio. The recent trend in applying these devices has been to vary the CD ratio based on the velocity to obtain more precision of control when moving at slow speeds. Example patents describing rate dependent adjustment of the CD ratio include: U.S. Pat. No. 5,191,641, Yamamoto, et al., issued Mar. 2, 1993, and U.S. Pat. No. 4,386,346, James L. Levine, issued May 31, 1983.
Input from a mouse is also often used to control a person's view of a 3D computer generated scene. Although there are a wide variety of mappings used to adjust one's view, one of the most widely used approaches, especially in first-person-shooter computer games, is to use the 2-DoF mouse input to control rotational components of the avatar's viewpoint. Side-to-side movements of the mouse adjust the yaw of the avatar's situated-heading, and front-to-back movements adjust the pitch of the avatar's head-pose. Notice that here a 2-DoF relative translational input is mapped to control two relative angular adjustments. The same methods of setting a CD ratio and varying the CD ratio in a rate dependent manner can also be applied to controlling the yaw and pitch of the viewpoint. Although a mouse could be used as an open loop control over distance (or angle) the remapping of linear distance covered based on velocity makes the mouse into a closed loop control.
Control over the pitch of the avatar's head-pose is treated separately from virtual locomotion, although it is as described below a key element of view control. On the other hand, control over the yaw of the situated-heading determines the heading and is thus an integral part of the virtual locomotion.
Keyboard and mouse controls are popular devices for controlling the avatar's viewpoint in first-person shooter video games because the precision of the mouse facilitates direct, precise, looking and aiming. The mouse is used to control the heading and elevation of the viewpoint. A set of four direction keys on the keyboard give the user a coarse form of directional control over the course-offset and rate control over the viewpoint's translation. Since a key press is mapped to a fixed velocity, the direction keys provide discrete rather than continuous control over the rate of displacement (the use taps the key slowly to move slowly and holds it down to move at the maximum velocity).
The keyboard and mouse control has similarities to a conventional gamepad control in that both use course-offset control mappings: turning the view changes the reference frame in which the direction of the course is interpreted. A fundamental difference is that the keyboard and mouse provides directional control over both steering and course-offset (versus the gamepad's rate control over steering and directional control over course-offset). The mouse's precise control of the viewpoint's yaw and pitch has made the keyboard and mouse popular for controlling first-person shooter games, since the user can quickly and accurately alter the view to steer the character and aim the character's weapon. Coordination between course and heading is still limited because they are controlled by different types of input devices (discrete keys versus continuous strokes). Keyboard and mouse controls are poorly suited for use with head-mounted displays because the user cannot see the keys of the keyboard.
Less Conventional Virtual Locomotion Controls
Another approach, designed to enhance a user's control over his viewpoint, is described in the patent: “Image Display Method and Apparatus with means for Yoking Viewpoint Orienting Muscles of a User” (U.S. Pat. No. 5,436,638 Bolas et al., issued Jul. 25, 1995). The control is intended to yoke the movement of a person's viewpoint orienting muscles to direct their viewpoint in VE. The user applies force to a spring centered input device specially designed to allow the user to apply forces from both sides of his body. These forces are yoked to provide a central measure of the user's muscle exertion. Up to 6-DoF of input can be expressed. The key idea behind this approach is that the movement of the viewpoint should react to the forces applied by the user—so as to turn or translate as the user's physical view would change if the forces were allowed to move the user's body rather than press against the device. To orient toward an area-of-interest on the left the user activates muscles that would be used to physically orient his body to the left and applies those forces to the device. The patent observed that: “The effect of virtual-motion of the viewpoint is compelling because the rate of motion increases with the forces applied.” This is a good example of a viewpoint control designed to be simple and natural to use. Notice, however, that even an advanced design like this one still relies on rate control to steer virtual motion. Thus ease of use is achieved at the cost of limiting the kinds of motion that can be easily executed and dependence on immediate visual feedback (closed loop control) to direct motion.
Another class of motion controls were developed to more closely simulate natural locomotion by having the user gesture with the legs to indicate the direction, rate, and extent with which virtual steps are taken. Such systems track the motion of the legs and other body segments in real time, and map localized leg movements into translational motion. Such a control allows a person to stand and walk in place to control the movement of their avatar's viewpoint through VE, and is described in U.S. Pat. No. 6,646,643, James N. Templeman, issued Nov. 11, 2003 (hereinafter “Templeman”). The range of motion is expressed by stepping to: advance, move obliquely, turn, move in a bladed posture (as when holding a rifle), and turret the upper body relative to the feet. People are proficient at combining these movements to perform complex maneuvers. A person's awareness of their dynamic posture and orientation in space facilitates this freedom of motion. This range of mobility is called upon to carry out the movements involved in infantry combat. The differences between motions expressed by leg-based gestural controls and motions expressed using vehicular and computer game input controls are presented.